Search engines are known. Search engines enable users to search for documents on the World Wide Web (WWW). A search engine searches documents for specified keywords and returns a list of the documents to a user probing for pertinent information. A search engine is a general class of programs.
Conventional search engines work by sending out a spider to fetch as many documents as possible. The term “search engine spider” can be used interchangeably with the term “search engine crawler.” A spider is a program that a search engine uses to seek out and index the information that it finds. Traditionally, the search engine spider “reads” the text on the web page, or collection of web pages, and records any hyperlinks it finds. Search engines can have spiders visit many sites in parallel, so that efficient cataloging will occur. Spiders visit sites that have been submitted by website owners as new or updated.
The spider method of cataloging information usually entails another program called an indexer which reads documents and creates a record based on the words contained in each document. Indexing facilitates finding pertinent references for searchers using a variety of keywords and gathering similar information under a single topic. Web indexing refers to various methods for indexing the contents of a website or of the Internet as a whole.